Trace formulas and dynamical zeta functions in the 

Nielsen theory 



In this paper we prove the trace formulas for the Reidemeister numbers of 
group endomorphisms in the following cases:the group is finitely generated and 
an endomorphism is eventually commutative; the group is finite ; the group 
is a direct sum of a finite group and a finitely generated free abelian group; 
the group is finitely generated, nilpotent and torsion free . These results had 
previously been known only for the finitely generated free abelian group. As a 
consequence, we obtain under the same conditions on the fundamental group 
of a compact polyhedron, the trace formulas for the Reidemeister numbers of 
a continuous map and under suitable conditions the trace formulas for the 
Nielsen numbers of a continuous map. The trace formula for the Reidemeister 
numbers implies the rationality of the Reidemeister zeta function. We prove 
the rationality and functional equation for the Reidemeister zeta function of 
an endomorphisms of finitely generated torsion free nilpotent group and of a 
direct sum of a finite group and a finitely generated free abelian group.We give 
a new proof of the rationality of the Reidemeister zeta function in the case when 
the group is finitely generated and the endomorphism is eventually commutative 
and in the case when group is finite. We give also another proof for the positivity 
of the radius of convergence of the Nielsen zeta function and propose an exast 
algebraic lower bound estimation for the radius. We connect the Reidemeister 
zeta function of an endom! orphism of a direct sum of a finite group and a 
finitely generated free abelian group with the Lefschetz zeta function of the 
unitary dual map, and as consequence obtain a connection of the Reidemeister 
zeta function with Reidemeister torsion. 



Introduction 

We assume everywhere X to be a connected, compact polyhedron and / : X — > X to 
be a continuous map. Taking a dynamical point on view,we consider the iterates of 
/.In the theory of discrete dynamical systems the following zeta functions are known: 
the Artin-Mazur zeta function 
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where F(f n ) is the number of isolated fixed points of / n ; the Ruelle zeta function |35 



co n- 1 



n 

K n=l xGFix(f n ) k=0 



where g : X — > (F is a weight function(if 5 = 1 we recover C/( z ))) the Lefschetz zeta 
function 

:= exp ^ 



vn=l 



where 



dimX 
k=0 

are the Lefschetz numbers of the iterates of /; reduced mod 2 Artin-Mazur and Lef- 



schetz zeta functions | 19| ; twisted Artin-Mazur and Lefschetz zeta functions ||20||, which 
have coefficients in the group rings ZjH of an abelian group H. 

The above zeta functions are directly analogous to the Hasse-Weil zeta function 
of an algebraic manifold over finite fields |39|].The Lefschetz zeta function is always 
rational function of z and is given by the formula: 

dimX 

L f (z)= J] det(£-/, fc .^) ( - 1)fc+1 . 

fc=0 

This immediately follows from trace formula for the Lefschetz numbers of the iterates 
of /.The Artin-Mazur zeta function has a positive radius of convergence for a dence set 
in the space of smooth maps of a compact smooth manifold to itself .Manning proved 
the rationality of the Artin-Mazur zeta function for diffeomorphisms of a compact 
smooth manifold satisfying Smale's Axiom A PC . 



The Artin-Mazur zeta function and its modification count periodic points of a map 
geometrically, the Lefschetz's type zeta functions do this algebraically (with weight 
given by index theory) Another way to count the periodic points is given by Nielsen 
theory. 

Let p : X — > X be the universal covering of X and / : X — > X a lifting of /, ie. 
pof = fop. Two liftings / and /' are called conjugate if there is a 7 G T = TTi(X) such 
that f = 70/0 7~ x . The subset p(Fix(f)) C Fix(f) is called the fixed point class of 
f determined by the lifting class [/] . A fixed point class is called essential if its index 
is nonzero. The number of lifting classes of / (and hence the number of fixed point 
classes, empty or not) is called the Reidemeister Number of /, denoted R{f)- This 
is a positive integer or infinity. The number of essential fixed point classes is called 
the Nielsen number of /, denoted by N(f). The Nielsen number is always finite. 
R(f) and N(f) are homotopy type invariants. In the category of compact, connected 
polyhedra the Nielsen number of a map is equal to the least number of fixed points 
of maps with the same homotopy type as /. Let G be a group and <fi : G — > G an 
endomorphism. Two elements a, a' G G are said to be — conjugate iff there exists 
7 G G such that a' = r y.a.(p( r y)~ 1 . The number of 0-conjugacy classes is called the 
Reidemeister number of 0, denoted by R(4>). 

If we consider the iterates of / and <p, we may define several zeta functions con- 



nected with Nielsen fixed point theory (see 0, [11], [13], [14], [Tj| ) . We assume throughout 
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this article that R{f n ) < oo and R(<f) n ) < oo for all n > 0. The Reidemeister zeta 
functions of / and <p and the Nielsen zeta function of / are defined as power series: 



oo T)( fn\ 



R f (z) := exp(E i/(/ 



n=l 71 



N f (z) := expf£ A ' (/ 



-z n 



n=l 



Rf(z) and Nf(z) are homotopy invariants. The Nielsen zeta function Nf(z) has a 
positive radius of convergence which has a sharp estimate in terms of the topological 
entropy of the map / [fLSl.In section 4 we propose another prove of positivity of the 



radius and give an exact algebraic lower estimation for radius using the Reidemeister 
trace formula for the generalised Lefschetz numbers. In section 7 we prove the same 
result for the radius of convergence of the minimal dynamical zeta function. 

An endomorphism : G — > G is said to be eventually commutative if there exists 
a natural number n such that the subgroup <p n (G) is commutative. A map / : X — > X 
is said to be eventually commutative if the induced endomorphism on fundamental 
group is eventually commutative. We begin the article by proving in section 1 the 
trace formula for the Reidemeister numbers in the following cases: 

(I) G is finitely generated and is eventually commutative. 

(II) G is finite 

(III) G is a direct sum of a finite group and a finitely generated free abelian group. 

(IV) G is a finitely generated torsion free nilpotent group. 

These result had previously been known only for the finitely generated free abelian 



group [FJ]. As a consequence, we obtain in section 2, under the same conditions (I) - 
(IV) on the fundamental group of X ,the trace formula for the Reidemeister numbers 
of a continuous map and in section 3 under suitable conditions the trace formulas for 
the Nielsen numbers of a continuous map. 

Trace formula for the Reidemeister numbers implies the rationality of the Reide- 
meister zeta function. In section 5 we prove the rationality and functional equation 
for the Reidemeister zeta function of an endomorphisms of finitely generated torsion 
free nilpotent group and of a direct sum of a finite group and a finitely generated free 
abelian group. We give also a new proof of the rationality of R^(z) in the case when G 
is finitely generated and is eventually commutative and in the case when G is finite. 

In section 7 we obtain an expression for the Reidemeister torsion of the mapping 
torus of the dual map of a endomorphism of a direct sum of a finite group and a 
finitely generated free abelian group, in terms of the Reidemeister zeta function of the 
endomorphism. The result is obtained by expressing the Reidemeister zeta function 
in terms of the Lefschetz zeta function of the dual map, and then applying the the- 
orem of D. Fried. What this means is that the Reidemeister torsion counts the fixed 
point classes of all iterates of map / i.e. periodic point classes of /.These result had 
previously been known for the finitely generated abelian groups and finite groups [|T6|| . 

In section 5.5 we describe some conjectures on how the Reidemeister zeta functions 
should look in general, largely in terms of the growth on the group. In particular 
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we expect that for polynomial growth groups some power of the Reidemeister zeta 
function is a rational function. 



We would like to thank J.Eichhorn , D. Fried, M.L.Gromov and B.B.Venkov for 
valuable conversations and comments. Alexander Fel'shtyn would like to thank Institut 
des Hautes Etudes Scientifiques for the kind hospitality and support. 

1 Trace formula for the Reidemeister numbers of 
a group endomorphism 

1.1 Pontryagin Duality 

Let G be a locally compact abelian topological group. We write G for the set of 
continuous homomorphisms from G to the circle U(l) = {z G (F: \z\ = 1}. This is 
a group with pointwise multiplication. We call G the Pontryagin dual of G. When 
we equip G with the compact-open topology it becomes a locally compact abelian 
topological group. The dual of the dual of G is canonically isomorphic to G. 

A continuous endomorphism / : G — > G gives rise to a continuous endomorphism 
f :G^G defined by 

fix) ■= X ° /• 

There is a 1-1 correspondence between the closed subgroups H of G and the quotient 
groups G/H* of G for which H* is closed in G. This correspondence is given by the 
following: 

H G/H*, 

H* := { X eG\H Cker X }. 

Under this correspondence, G/H* is canonically isomorphic to the Pontryagin dual 
of H. If we identify G canonically with the dual of G then we have H** = H. If G is 
a finitely generated abelian group then a homomorphism x '■ G — > U(l) is completely 
determined by its values on a basis of G, and these values may be chosen arbitrarily. 
The dual of G is thus a torus whose dimension is equal to the rank of G. 
If G = TZ/nZi then the elements of G are of the form 

x — > e « 

with y E {1, 2, . . . , n}. A cyclic group is therefore (uncanonically) isomorphic to itself. 

The dual of G\ © Gi is canonically isomorphic to G\ @G-i- From this we see that 
any finite abelian group is (non-canonically) isomorphic to its own Pontryagin dual 
group, and that the dual of any finitely generated discrete abelian group is the direct 
sum of a Torus and a finite group. 



Proofs of all these statements may be found, for example in [34| . We shall require 
the following statement: 

Proposition 1 Let : G — > G be an endomorphism of an abelian group G. Then the 



kernel ker 



G^G 



is canonically isomorphic to the Pontryagin dual of Coker 



4 



Proof We construct the isomorphism explicitly. Let \ be in the dual of Coker (0 : 
G — > G). In that case x is a homomorphism 

X ■ G/lm (0) — £7(1). 

There is therefore an induced map 

X--G-^U(1) 

which is trivial on Im (0). This means that x ° is trivial, or in other words 0(x) 
is the identity element of G. We therefore have x G ker(0). If on the other hand we 
begin with x G ker(0), then it follows that x is trivial on Im 0, and so x induces a 
homomorphism 

X : G/lm (0) — C/(l) 
and x is then in the dual of Coker 0. The correspondence x *-* X is clearly a bijection. 

1.2 Eventually commutative endomorphisms 

1.2.1 Trace formula for the Reidemeister numbers of eventually commu- 
tative endomorphisms. 

An endomorphism : G — > G is said to be eventually commutative if there exists a 
natural number n such that the subgroup 4> n {G) is commutative. If is an endomor- 
phism of an abelian group G then x and y are — conjugate iS x — y = 0(g) — g for 
some g G G. Therefore -R(0) is the number of cosets of the image of the endomorphism 
(0 - 1) : G -> G.Then 

i?(0) = #Coker (1-0). 

We are now ready to compare the Reidemeister number of an endomorphism with 
the Reidemeister number of i?i(0) : H\{G) — > Hi(G), where Hi = H 1 P is the first 
integral homology functor from groups to abelian groups. 

Lemma 1 ( ||26|| ) If : G — ► G is eventually commutative, then 

i?(0) = i?(#!(0) = #Coker (1 - ^(0)) 

This means that to find out about the Reidemeister numbers of eventually com- 
mutative endomorphisms, it is sufficient to study the Reidemeister numbers of endo- 
morphisms of abelian groups. For the rest of this section G will be a finitely generated 
abelian group. 

Lemma 2 ( ||14|| ) Let : ZZ k — > *%j k be a group endomorphism. Then we have 

R(4>) = (-ir + ^(-l) l Tr (AV). (1) 

i=0 

where p the number of /i G Spec such that a < —1, and r the number of real 
eigenvalues of whose absolute value is > 1. A 1 denotes the exterior power. 
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Proof Since Z k is abelian, we have as before, 

R((f)) = #Coker (1-0). 

On the other hand we have 

#Coker (1-0) =| det(l -0) |, 

and hence R(4>) = (— l) r+p det(l — 0) ( complex eigenvalues contribute nothing to the 
signdet(l — 0) since they come in conjugate pairs and (1 — A)(l — A) =| 1 — A | 2 > 0). 
It is well known from linear algebra that det(l — 0) = Y,i=o(~ l)*Tr (A*0). From this 
we have the trace formula for Reidemeister number. 

Now let be an endomorphism of finite abelian group CLet V be the complex 
vector space of complex valued functions on the group CThe map induces a linear 
map A : V — > V defined by 

A(f) :=/o0. 



Lemma 3 Let : G — > G be an endomorphism of a finite abelian group G. Then we 
have 

R{<t>) = TrA (2) 

We give two proofs of this lemma in this article. The first proof is given here and the 
second proof is a special case of the proof of theorem 4 

Proof We shall calculate the trace of A in two ways. The characteristic functions 
of the elements of G form a basis of V,and are mapped to one another by A(the map 
need not be a bijection). Therefore the trace of A is the number of elements of this 
basis which are fixed by A. On the other hand, since G is abelian, we have, 

R((f)) = #Coker (1-0) 
= #G/#Im(l-0) 
= #G/#(G/ker(l-0)) 
= #G/(#G/#ker(l-0)) 
= #ker(l-0) 
= #Fix(0) 

We therefore have R((f>) = #Fix (0) = TrA . 

For a finitely generared abelian group G we define the finite subgroup TorsG to be 
the subgroup of torsion elements of G. We denote the quotient G°° := G/TorsG. The 
group G°° is torsion free. Since the image of any torsion element by a homomorphism 
must be a torsion element, the endomorphism : G — > G induces endomorphisms 



Ltor 



TorsG — ► TorsG, 0°° : G°° — ► G c 



As above, the map tor induces a linear map A : V — > V, where V be the complex 
vector space of complex valued functions on the group TorsG. 
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Theorem 1 If G is a finitely generated abelian group and an endomorphism of G 
. Then we have 

k 

R((j)) = (-l) r +P^(-l)*Tr (A>°° ® A). (3) 

where k is rgG°°, p the number of // G Spec 0°° suc/i £/iat /x < — 1, and r i/ie number 
of real eigenvalues of 0°° whose absolute value is > 1. 

PROOF By proposition 1, the cokernel of (1 — 0) : G — > G is the Pontrjagin dual 
of the kernel of the dual map (1 — 0) : G — > G. Since Coker (1 — 0) is finite, we have 

#Coker (1 - 0) = #ker (1^0). 

The map 1 — is equal to 1 — 0. Its kernel is thus the set of fixed points of the map 
: G — > G. We therefore have 



i?(0) = #Fix (0 : G -> GJ (4) 

The dual group of G°° is a torus whose dimension is the rank of G. This is canonically 
a closed subgroup of G. We shall denote it G . The quotient G/G is canonically 
isomorphic to the dual of TorsG. It is therefore finite. From this we know that G is 
a union of finitely many disjoint tori. We shall call these tori Gq, . . . ,G t - 

We shall call a torus G, periodic if there is an iteration s such that s (Gj) C Gj. 
If this is the case, then the map s : G» — > G» is a translation of the map s : Go — > Go 
and has the same number of fixed points as this map. If s (Gj) ^ Gj then s has no 
fixed points in G«. From this we see 

#Fix (0 : G - G) = #Fix (0 : G - G ) x #{G, | 0(G,) c G,}. 
We now rephrase this 

#Fix (0 : G -> G) 
= #Fix (0^ : G ^ G ) x #Fix : G/(G ) ^ G/(G )) . 

From this we have product formula for Reidemeister numbers 

R(<f>) = J R(0 oo ). J R(0 tors ). 
The trace formula for i?(0) follow from the previous two lemmas and formula 

Tr(A>°°).Tr(A) = Tr(A i (j) ° ® A). 



In the paper |T6[ we have connected Reidemeister number of endomorphism with 
Lefschetz number of the dual map. From this we have following trace formula 

Theorem 2 ([|16||) Let <ft : G ^ G be an endomorphism of a finitely generated abelian 
group. Then 

dimG* 

i?(0) =| L(0) |= {-1Y+P J2 (- 1 )" Tr : H k(G; Q) -> H k (G; Q)] (5) 

where is the continuous endomorphism ofG defined in §2.2 and L((f>) is the Lefschetz 
number of thought of as a self-map of the topological space G and r and p are the 
constants described in theorem 1. If G is finite then this reduces to 

R{(j>) = L(0) = Tr [0,o : H (G; Q) -> H (G; Q) 
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1.3 Endomorphisms of finite groups 

In this section we consider finite non-abelian groups. We shall write the group law 
multiplicatively. We generalize our results on endomorphisms of finite abelian groups 
to endomorphisms of finite non-abelian groups. We shall write {g} for the 0-conjugacy 
class of an element g G G. We shall write < g > for the ordinary conjugacy class of g 
in G. We first note that if is an endomorphism of a group G then <j) maps conjugate 
elements to conjugate elements. It therefore induces an endomorphism of the set of 
conjugacy classes of elements of G. If G is abelian then a conjugacy class of element 
consists of a single element. The following is thus an extension of lemma 3: 

Theorem 3 ( [[Tj| ) Let G be a finite group and let <ft : G —>■ G be an endomorphism. 
Then R((f>) is the number of ordinary conjugacy classes < x > in G such that 

< (f)(x) >=< x > . 
Proof From the definition of the Reidemeister number we have, 

m = E 1 

{9} 

where {g} runs through the set of 0-conjugacy classes in G. This gives us immediately 

{9} xe{g} ^tyj 

= E— ■ 

We now calculate for any x G G the order of {x}. The class {x} is the orbit of x 
under the G-action 

(g,x) 1 — > gx^gY 1 . 
We verifty that this is actually a G-action: 

(id,x) 1 — > id.x.cl){id)~ l 

(9192, x) 1 — ► g^.x.^g^y 1 

= 9i92-x.((j)(gi)(j)(g2)y 1 
= 9i92-x.<p{92y 1 <P{9iy 1 

= 9i(92-x.(j)(g2y 1 )(j)(giy 1 - 
We therefore have from the orbit-stabilizer theorem, 



#{*} 



#{<? G G I gx<p(g)~ l = x}' 
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The condition gx<p(g) 1 = x is equivalent to 

x' 1 gx^g)' 1 = 1 <^> x' x gx = <j>(g) 

We therefore have 

Changing the summation over x to summation over g, we have: 

" gGG 

If < 0(g) >t^< <? > then there are no elements x such that x~ x gx = (f>(g). We 
therefore have: 

R &) = irr E #{^G|i> = ^)}. 

TT^ 9 6G such that 
<<Kg)> = <9> 

The elements a; such that x~ x gx = (f>(g) form a coset of the subgroup satisfying 
x~ x gx = g. This subgroup is the centralizer of g in G which we write C(g). With this 
notation we have, 

R(<l>) = 4^ E #C{g) 

TT*- 7 geG such that 
<0(9)> = <9> 

= ^77 E #<^>.#C(^). 

TT 1 - 7 <j>CG such that 
<<Hs)> = <S> 

The last identity follows because CQi^gh) = h^ 1 C(g)h. From the orbit stabilizer 
theorem, we know that # < g > .#C(g) = #G. We therefore have 

W) = #{< 9>cG|< 0(5) >=< g >}• 

Let be the complex vector space of complex valued class functions on the group 

G.A class function is a function which takes the same value on every element of a 
usual congruency class. The map induces a linear map B : W — > W defined by 

B(f) :=/o0. 



Theorem 4 Lei <p : G ^ G be an endomorphism of a finite group G. Then we have 

R{4>) = TrB (6) 

Proof We shall calculate the trace of B in two ways. The characteristic functions 
of the congruency classes in G form a basis of Vr,and are mapped to one another by 
£?(the map need not be a bijection). Therefore the trace of B is the number of elements 
of this basis which are fixed by B. By theorem 3, this is equal to the Reidemeister 
number of 0. 
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1.4 Endomorphisms of the direct sum of a free abelian and 
a finite group 

In this section let F be a finite group and r a natural number. The group G will be 
G = Zi k © F.The torsion elements of G are exastly the elements of the finite, normal 
subgroup F.For this reason we have 0(F) C F.Let <f> tors : F — > F be the restriction of 
to F, and let 0°° : G/F — > G/F be the induced map on the quotient group. 



Lemma 4 In i/je notation described above 

#(0) = F(0°°).F(0 tors ) 

Let be the complex vector space of complex valued class functions on the group 
F.The map induces a linear map B : W — > W defined as above. 

Theorem h If G is the direct sum of a free abelian and a finite group and an 
endomorphism of G . Then we have 

k 

i?(0) = (-l) r +P^(-l) i Tr (A>°° <g> 5). (7) 

where k is rg{G/F), p the number of \i e Spec 0°° snc/i i/iai // < — 1, and r £/ie 
number of real eigenvalues of (ft 00 whose absolute value is > 1. 

PROOF Theorem follows from lemmas 2 and 4 , theorem 4 and formula 

Tr(A> 00 ).Tr( J B) = Tr(A>°° <g> B). 



1.5 Endomorphisms of nilpotent groups 

In this section we consider finitely generated torsion free nilpotent group T.It is well 
known |29|] that such group T is a uniform discrete subgroup of a simply connected 
nilpotent Lie group G (uniform means that the coset space G/Y is compact). The coset 
space M = G/Y is called a nilmanifold. Since Y = ivi(M) and M is a K(Y, 1), every 
endomorphism : Y — > Y can be realized by a selfmap / : M — > M such that /* = 
and thus R(f) = -R(0).Any endomorphism : Y — > Y can be uniquely extended to 
an endomorphism F : G — > G.Let F : G — > G be the corresponding Lie algebra 
endomorphism induced from F. 

Theorem 6 If G is a finitely generated torsion free nilpotent group and an endo- 
morphism of G . Then 

m 

i?(0) = (-if+p jT(-l) < T!r A'F, (8) 

where m is rgY = dimM, p the number o/juS Spec F such that \x < — 1, and r 
the number of real eigenvalues of F whose absolute value is > 1. 
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Proof: Let / : M — > M be a map realizing on a compact nilmanifold M of 
dimension m.We suppose in this article that the Reidemeister number R(f) = R{4>) 
is finite. The finiteness of R(f) implies the nonvanishing of the Lefschetz number L(f) 
JTTfl.A strengthened version of Anosov's theorem |T[j is proven in |n|] which states, 
in particular, that if L(f) 7^ than iV(/) = \L(f)\ = R(f ).lt is well known that 
L(f) = det(F - 1) 0.From this we have 

m 

R{#) = R(f) = \L(f)\ = I det(l-F)| = (-l)^det(l-F) = A*F. 



i=0 



1.6 The trace formulas and group extensions. 

Suppose we are given a commutative diagram 

G ^ G 

IP _ IP (9) 
G -^-> G 

of groups and homomorphisms. In addition let the sequence 

O^H^G^G^O (10) 
be exact. Then restricts to an endomorphism |#: H — > if. 

Definition 1 T7ie s/iort exaci sequence (16) of groups is said to have a normal split- 
ting if there is a section a : G — > G of p such that Im a = o~{G) is a normal subgroup 
of G. An endomorphism <ft : G —>■ G is said to preserve this normal splitting if <fi 
induces a morphism of (16) with <fi(a(G)) C a{G). 

In this section we study the relation between the traces formulas of type (7) for 
Reidemeister numbers R(<f)), R(<fi) and R(<p |#). 

Theorem 7 Let the sequence (14) have a normal splitting which is preserved by : 
G — ► G. If we have a trace formulas of the type (7) for i?(0) and i?(0 \h) then we 
have a trace formula of the same type for R(<f>). 

Proof From the assumptions of the theorem it follows that 

i?(0) = R@).R{<t> \h) (see @). 

Trace formula for R(<f>) now follows from linear algebra formula: 

TrA.TrB = TrA <g> B. 

1.6.1 Direct Sums 

If G = G\ © G2 is a direct sum and if 0(Gj) C Gi for i = 1, 2 then it has been shown 



sec 



23]) that R(4>) = -R(0i) -R{4>2) where 0j is the restriction of to Gj.So if we have 
the trace formula of type (7) for R{4>\) and R(<p2) then we have the trace formula for 
R{4>). 
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2 Trace formulas for Reidemeisters numbers of a 
continuous map. 

Let / : X — > X be given, and let a specific lifting / : X — > X be chosen as reference. 
Let T be the group of covering translations of X over X. Then every lifting of / can 
be written uniquely as 7 o /, with 7 G T. So elements of T serve as coordinates of 
liftings with respect to the reference /. Now for every 7 G T the composition / o 7 is 
a lifting of / so there is a unique 7' G T such that 7' o / = / o 7. This correspondence 
7 — > 7' is determined by the reference /, and is obviously a homomorphism. 

Definition 2 The endomorphism /* : T — > T determined by the lifting f of f is 
defined by 

f*(l) /= / 07. 

It is well known that T = 7Ti(X). We shall identify n = ni(X,x ) and T in the 
following way. Pick base points x G X and 5 £ P _1 ( a; o) C X once and for all. Now 
points of X are in 1-1 correspondence with homotopy classes of paths in X which start 
at Xq: for x G X take any path in X from x to 5 and project it onto X; conversely 
for a path c starting at x , lift it to a path in X which starts at x , and then take its 
endpoint. In this way we identify a point of X with a path class < c > in X starting 
from Xq. Under this identification, x — < e > is the unit element in tti(X,xo). The 
action of the loop class a =< a >G 717 (X, x ) on X is then given by 

a =< a >:< c >^ a.c =< a.c > . 

Now we have the following relationship between f^-.n^n and 

/* : 7ri(X,Xo) — ► 7Ti(X,/(x )). 

Lemma 5 Suppose f(x ) =< w >. Then the following diagram commutes: 

7Ti(X,X ) — -> 7Ti(X, /(Xo)) 

/* \ I w * 

7Ti(X, X ) 

where w* is isomorphism induced by the path w. 
In other words, for every a =< a >G 717 (X, x ) , we have 

/*(< a >) =< w(f o a)w _1 > 



Remark 1 In particular, if x G p(Fix(f) and x G Fix(f) ,then /* = 

Lemma 6 Lifting classes of f are in 1-1 correspondence with f*-conjugacy classes in 
n, the lifting class [7 o /] corresponding to the f^-conjugacy class 0/7. We therefore 
have R(f) = R(f*). 
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We shall say that the fixed point class p{Fix{ r y of)), which is labeled with the 
lifting class [70/], corresponds to the /*-conjugacy class of 7. Thus /*-conjugacy 
classes in it serve as coordinates for fixed point classes of /, once a reference lifting / 
is chosen. 

Let us consider a homomorphisms from it sending an /^-conjugacy class to one 
element: 

Lemma 7 ( ||26|| ) The composition rj o $, 

it = tti(X, x ) H X (X) Coker H X (X) ^ H^X) 

where 9 is abelianization and rj is the natural projection, sends every f*-conjugacy 
class to a single element. Moreover, any group homomorphism ( : 71 — > G which 
sends every f^-conjugacy class to a single element, factors through 77 o 6. 



Definition 3 A map f : X —>■ X is said to be eventually commutative if there exists 
an natural number n such that f^(7Ci(X,x )) (c tti{X, f n (xo))) is commutative. 

By means of Lemma 5, it is easily seen that / is eventually commutative iff /* is 
eventually commutative (see p6|). 

Now using lemma 6 we may apply all theorems of §1 to the Reidemeister numbers 
of continuous maps. 



2.1 Trace formulas and Serre bundles. 

For example, let us consider topological counterpart of theorem 7. 

Let p : E — > B be a Serre bundle in which E, B and every fibre are connected, 
compact polyhedra and Ff, = p~ x (b) is a fibre over b G B. A Serre bundle p : E — > B 
is said to be (homotopically) orientable if for any two paths w, w' in B with the same 
endpoints w(0) = w'(0) and w(l) = w'(l), the fibre translations r w , t w > : F w ^ — > F w ^ 
are homotopic. A map / : E — > E is called a fibre map if there is an induced map 
/ : B —>■ B such that p o f — f o p. Let p : E — > B be an orientable Serre bundle and 
let / : E —>■ E be a fibre map. Then for any two fixed points b, b 1 of / : B — > B the 
maps fb = f |jr 6 and fy = f \p b , have the same homotopy type; hence they have the 
same Reidemeister numbers R(fb) = R(fb') HI- 



Theorem 8 Suppose that f : E — > E admits a Fadell splitting in the sense that for 
some e in Fix / and b = p(e) the following conditions are satisfied: 

1. the sequence 

— > 7n(F 6 , e) 7n(£, e) ^ m(B, e) — > 

is exact, 

2. p* admits a right inverse (section) a such that Im o is a normal subgroup of 
7Ti(E,e) and /*(Im a) C Im a. 

If we have a trace formulas of the type (7) for R(f) and R(fb) then we have a trace 
formula of the same type for R(f). 
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3 Trace formulas for the Nielsen numbers 



3.1 The Jiang subgroup and trace formula 

From the homotopy invariance theorem (see f26|| ) it follows that if a homotopy {ht} : 
/ = g : X — > X lifts to a homotopy {h t } : / = g : X — > X, then we have 
Index (/, p(Fix /)) = Index(g,p(Fia; #)). Suppose { /i f }_ is a cyclic homotopy : 
f — f; then this lifts to a homotopy from a given lifting / to another lifting /' = aof, 
and we have 



Index(/, p(Fix /)) = Index(/, p(Fix ao /)). 

In other words, a cyclic homotopy induces a permutation of lifting classes (and hence 
of fixed point classes); those in the same orbit of this permutation have the same 
index. This idea is applied to the computation of N(f). 

Definition 4 The trace subgroup of cyclic homotopies (the Jiang subgroup) 1(f) C tt 
is defined by 



1(f) 



a £ 7r 



there exists a cyclic homotopy 
{ht} : / = /which lifts to 
{h t }:f = aof 



(see $QJ) 



Let Z(G) denote the centre of a group G, and let Z(K, G) denote the centralizer 
of the subgroup K C G. The Jiang subgroup has the following properties: 1.1(f) C 
Z(j*(7r),7r); 

2./(zd^) C Z(tt); 3.J(sf) C /0 o /); 4.g*(I(f)) C /(§ o /); 5./(z^) C /(/). 

The class of path-connected spaces X satisfying the condition I(id^) = tt = 
tti(X,x ) is closed under homotopy equivalence and the topological product oper- 
ation, and contains the simply connected spaces, generalized lens spaces, if -spaces 
and homogeneous spaces of the form G/Gq where G is a topological group and Go a 



subgroup which is a connected, compact Lie group (for the proofs see [E| 



From theorem 1 and results of Jiang [26| it follows: 



Theorem 9 Suppose that there is an integer m such that fl 
0. Then 



(n)cl(f m ) andL(f)^ 



N(f) = R(f) 



■ir^(-i)^(A ! / lt °°®A). 



(11) 



where k is rgHi(X, 2Z)°° ,A is linear map on the complex vector space of complex 
valued functions on the group TorsHi(X, 2Z), p the number of \i 6 Spec such that 
H < —1, and r the number of real eigenvalues of f™ whose absolute value is > 1. 



Proof We have /*(7r) C I(f)(see J26|) . For any a e it, p(Fix aof) = p(Fix f*(a)of) 
by the fact (see [p6[| ) that a and f*(a) are in the same /*-conjugacy class. 

Since /*(7r) C /(/), there is a homotopy {h t } : / = / which lifts to {h t } : 
/ = o /. Hence Index (/, p(Fix /)) = Index (/, p(Fix a o /)). Since a G 7r is 

arbitrary, any two fixed point classes of / have the same index. It immediately follows 
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that L(f) = implies N(f) = and L(f) ^ implies N(f) = R(f). By property 1, 
f{ii) C 1(f) C Z(f*(ir), tt), so /*(vr) is abelian. Hence /* is eventually commutative 
and N(f) = R(f) = R(f*) = R(fu). The result now follows from theoreml. 

Example 1 Let f : X — > X be a hyperbolic endomorphism of torus T k .Then Hi(X, 2Z) 
is torsion free and 

N(f) = R(f) = (-ir + ^(-l)*Tr (A%). (12) 

i=0 

3.2 Polyhedra with finite fundamental group. 

For a compact polyhedron X with finite fundamental group, tti(X), the universal cover 
X is compact, so we may explore the relation between L(f) and Index (p(Fix f)). 

Definition 5 The number = j^Fix f*, defined to be the order of the finite 

group Fix /*, is called the multiplicity of the lifting class [f], or of the fixed point 
class p(Fix f ). 

Lemma 8 ([]2B|]) 

L(f) = M[/]).Index (f,p(Fixf)). 

Let W be the complex vector space of complex valued class functions on the funda- 
mental group 7r.The map /* induces a linear map B : W W defined by 

B(f):=foU 

Theorem 10 Let X be a connected, compact polyhedron with finite fundamental 
group it. Suppose that the action of it on the rational homology of the universal cover 
X is trival, i.e. for every covering translation a 6 n , a* = id : H^X, Q) — > H*(X, Q). 
Let L(f) ^ O.Then 

N{f) = R(f) = Tr B, (13) 

Proof Under our assumption on X, any two liftings / and a o f induce the same 
homology homomorphism if*(X,($) — > H*(X,Q), and have thus the same value of 
L(f). From Lemma 8 it follows that any two fixed point classes / are either both 
essential or both inessential. Since L(f) ^ there is at least one essential fixed point 
class of /. Therefore all fixed point classes of / are essential and N(f) = R(f). The 
formula for N(f) follows now from theorem 4 

Lemma 9 Let X be a polyhedron with finite fundamental group n and let p : X — > X 
be its universal covering. Then the action of n on the rational homology of X is trivial 
iffH*(X;Q)^H*(X;Q). 

Corollary 1 Let X be a compact 1-connected polyhedron which is a 

rational homology n-sphere, where n is odd. Let tt be a finite group acting freely 
on X and let X = X/tt. Then theorem 10 applies. 
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Proof The projection p : X — > X = X /it is a universal covering space of X. For 
every a E tt, the degree of a : X — > X must be 1, because L(a) = (a has no fixed 
points). Hence a* = id : i/*(X; Q) — > H*(X; Q). 

Corollary 2 If X is a closed 3-manifold with finite it, then theorem 10 applies. 

Proof X is an orientable, simply connected manifold, hence a homology 3-sphere. 
We apply corollary . 

Corollary 3 Let X = L(m, qi, . . . , q r ) be a generalized lens space and f : X — > X a 
continuous map with = d where d ^ 1. Then theorem 10 applies. 

Proof By corollary 1 we see that theorem 10 applies for lens spaces. Since 
7Ti(X) = 2Z/mffl, the map / is eventually commutative. A lens space has a structure 
as a CW complex with one cell e, in each dimension < % < 21 + 1. The boundary 
map is given by de^k = m-^k-i f° r even cells, and <9e2/c+i — for odd cells. From this 
we may calculate the Lefschetz numbers: L(f) = 1 — d^ l+1 ^ ^ 0. 

3.3 Other special cases 
3.3.1 Self-map of a nilmanifold 

Theorem 6 implies 

Theorem 11 Let f be any continious map of a nilmanifold M to itself. If R(f) is 
finite then 

m 

N(f) = R(f) = (-lr+^-iyTr A*F , (14) 

i=0 

where F , m,r, and p are the same as in theorem 6 



3.3.2 Pseudo-Anosov homeomorphism of a compact surface 

Let X be a compact surface of negative euler characteristic and / : X — > X is a 
pseudo-Anosov homeomorphism,i.e. there is a number A > 1 and a pair of trans- 
verse measured foliations (F s ,fi s ) and (F u ,fi u ) such that /(F s ,/i s ) = (F s ,j^ s ) and 
f(F u ,fi u ) = (F u , A/i"). Fathi and Shub |J has proved the existence of Markov parti- 
tions for a pseudo-Anosov homeomorphism. The existence of Markov partitions implies 
that there is a symbolic dynamics for (X,f). This means that there is a finite set N, 
a matrix A = (di^a^eNxN with entries or 1 and a surjective map p : Q — > X, where 

Q = {(x n ) nez: : a XnXn+1 = l,n G %} 

such that p o cr = fop where a is the shift (to the left) of the sequence (x n ) of 
symbols. We have first 0: 

#Fixo n = Tr A n . 

In general p is not bijective.The non-injectivity of p is due to the fact that the rectan- 
gles of the Markov partition can meet on their boundaries. To cansel the overcounting 



of periodic points on these boundaries,we use Manning's combinatorial arguments [|30 
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proposed in the case of Axiom A diffeomorphism (see also p2[ ) .Namely, we construct 
finitely many subshifts of finite type 0£,i = 0, l,..,m, such that o"o = <J, the other 
shifts semi-conjugate with restrictions of / |32|] ,and signs e« G { — 1,1} such that for 
each n 

m m 

#Fixf n = 5>.#Fix< = E^r Al 

i=0 i=0 

where is transition matrix, corresponding to subshift of finite type a,. For pseudo- 
Anosov homeomorphism of compact surface N(f n ) = #Fix(f n ) for each n > o [[STJ.So 
we have following trace formula for Nielsen numbers 

Theorem 12 Let X be a compact surface of negative euler characteristic and f : 
X —>■ X is a pseudo-Anosov homeomorphism. Then 

m 

N(f n ) = ]>>Tr Al 
3.3.3 Homeomorphisms of a hyperbolic 3-manifolds 



Theorem 13 Jj?q/ Suppose M is a orientable compact connected 3-manifold such that 
intM admits a complete hyperbolic structure with finite volume and f : M —>■ M is 
orientation preserving homeomorphism. Then 

dim M 

N(f) = L(f) = (- 1 )" Tr [/•* : H k (X;Q) - H k (X;Q) 

k=0 

4 The Reidemeister trace formula for generalised 
Lefschetz numbers 

The results of this section are well known(see [Q,[]3^,||],f27|]).We shall use this results 



later in section to estimate the radius of convergence of the Nielsen zeta function. The 
fundamental group 7r = iri(X, x ) splits into /*-conjugacy classes. Let 717 denote the set 
of jVconjugacy classes, and % pif denote the abelian group freely generated by iif .We 
will use the bracket notation a —>■ [a] for both projections ir — > TTf and Zti — > /Z-kj. 
Let x be a fixed point of /.Take a path c from x to x.The /*-conjugacy class in n 
of the loop c.(f o c)" 1 , which is evidently independent of the choice of c, is called the 
coordinate of x.Two fixed points are in the same fixed point class F iff they have the 
same coordinates. This /*-conjugacy class is thus called the coordinate of the fixed 
point class F and denoted cd n (F, f) (compare with discription in section 2). 



The generalised Lefschetz number or the Reidemeister trace is defined as 



LAf) ■= E»^>/)-^(f./) e %*f, (15) 

F 

the summation being over all essential fixed point classes Fof /The Nielsen num- 
ber N(f) is the number of non-zero terms in L n (f), and the indices of the essential 
fixed point classes appear as the coefficients in L n (f). This invariant used to be called 
the Reidemeister trace because it can be computed as an alternating sum of traces 



on the chain level as follows (||33|,|38l) . Assume that X is a finite cell complex and 
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/ : X — > X is a cellular map. A cellular decomposition e d - of X lifts to a 7r-invariant 
cellular structure on the universal covering X. Choose an arbitrary lift for each 

.They constitute a free ^7r-basis for the cellular chain complex of X.The lift / of / 
is also a cellular map. In every dimension d, the cellular chain map / gives rise to a 
^7r-matrix Fd with respect to the above basis, i.e Fd = (ay) if fief) = J2j ay e^, where 
ay G ^7r.Then we have the Reidemeister trace formula 

LM = E(-!)1 T ^] e Zitf. (16) 



4.1 The mapping torus approach to the Reidemeister trace 
formula. 

Now we describe alternative approach to the Reidemeister trace formula proposed 



recently by Jiang ^7|. This approach is useful when we study the periodic points of 
/,i.e. the fixed points of the iterates of /. 

The mapping torus Tf of / : X — > X is the space obtained from X x [o, oo) by 
identifiing (x, s + 1) with (f(x), s) for all x G X, s G [0, oo).On T) there is a natural 
semi-flow <fi : Tf x [0, oo) — > Tf,<f) t (x,s) = (x, s + t) for all £ > O.Then the map 
/ : X — > X is the return map of the semi-flow 0.A point X G X and a positive 
number r > determine the orbit curve <fi( x , T ) := 0t( x )o<t<T m -^7- 

Take the base point Xo of X as the base point of Tf.lt is known that the funda- 
mental group H := 7Ti(Tf,xo) is obtained from 7r by adding a new generator z and 
adding the relations z~ x gz = f*(g) for all g E n = 7r 1 (X, x ).Let H c denote the set 
of conjugacy classes in H . Let ZH be the integral group ring of H, and let /ZH C be 
the free abelian group with basis H c .We again use the bracket notation a — > [a] for 
both projections H — > H c and ZH — > /ZH C . If F n is a fixed point class of f n , then 
f(F n ) is also fixed point class of f n and ind(f(F n ),f n ) = ind(F n ,f n ). Thus / acts 
as an index-preserving permutation among fixed point classes of / n .By definition, an 
n-orbit class O n of / to be the union of elements of an orbit of this action. In other 
words, two points x, x' G Fix(f n ) are said to be in the same n-orbit class of / if and 
only if some f l (x) and some f^{x') are in the same fixed point class of / n .The set 
Fix(f n ) splits into a disjoint union of n-orbits classes. Point x is a fixed point of f n 
or a periodic point of period n if and only if orbit curve <f>(x,n) is a closed curve. The 
free homotopy class of the closed curve 4>{x,n) will be called the H -coordinate of point 
x, written cdn(x, n) = [4>( x ,n)] G H c .lt follows that periodic points x of period n and 
x' of period n' have the same if-coordinate if and only if n = n' and x,x' belong to 
the same n-orbits class of /. Thus it is possible equivalently define x, x' G Fixf n to 
be in the same n-orbit class if and only if 



Recently, Jiang |27| has considered generalised Lefschetz number with respect to 

H 

L H (f n ) ■= E ind (° n ' f n )-cd H (O n ) G 2ZH C , (17) 
and proved following trace formula: 

MD = T,(-l) d [Tr{zF d ) n ] G ZH e , (18) 
d 

where F^ be ^7r-matrices defined in (16) and zF^ is regarded as a /ZH-m&trix. 
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5 The Reidemeister zeta function of a group endo- 
morphism 

PROBLEM. For which groups and endomorphisms is the Reidemeis- 
ter zeta function a rational function? When does it have a functional 
equation? Is R^z) an algebraic function? 

5.1 Reidemeister zeta functions of eventually commutative 
endomorphisms . 

As we remarked in section 1 to find out about the Reidemeister zeta functions of 
eventually commutative endomorphisms, it is sufficient to study the zeta functions 
of endomorphisms of abelian groups. For the rest of this section G will be a finitely 
generated abelian group. 

Theorem 14 Let G is a finitely generated abelian group and an endomorphism of 
G . Then R^{z) is a rational function and is equal to 



where matrix A is defined in lemma 3, a = (—l) p ,p , r and k are constants described 
in theorem 1. 



Proof If we repeat the proof of the theorem 1 for <\> n instead we receive that 
R(<f) n ) = R(((f) OD ) n .R(((p tor ) n ) .From this and lemmas 2 and 3 we have the trace formula 
for R((f) n y. 




(19) 



R(<p n ) 



k 

r+pn ' 
i=0 



l) l Tr A l (0°°) n .Tr A 



n 



k 




l) ir Tr (A*(0°°) n ® A n ) 



i=0 




l) ir Tr (A>°° ® A) n . 



We now calculate directly 
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5.2 Endomorphisms of finite groups 

Theorem 15 Let (f> be an endomorphism of a finite group G. Then R<j>(z) is a rational 
function and given by formula 

R ^ = asoW < 20) 

Where B is defined in theorem 4 

Proof From theorem 4 it follows that R(<f) n ) = Tr B n for every n > o.We now 
calculate directly 

= exp (Tr {-log{\ - Bz))) 



det(l - Bz)' 

5.3 Endomorphisms of the direct sum of a free abelian and 
a finite group 

Theorem 16 Let G is the direct sum of free abelian and a finite group and an 
endomorphism of G .If the numbers R((p n ) are all finite then R^{z) is a rational 
function and is equal to 

R^z) = \J[ det(l - A>°° <g) B.a.z) { - ir+1 j (21) 

where matrix B is defined in theorem 4, o = (—^) P ,P , t and k are constants described 
in theorem 5. 

Proof From lemma 4 it follows that R(4> n ) = R(((p°°) n .R((4> tor ) n ) . From now on 
the proof repeat the proof of the theorem 14. 

5.4 Endomorphisms of nilpotent groups 

Theorem 17 If G is a finitely generated torsion free nilpotent group and an endo- 
morphism of G .Then R<f>(z) is a rational function and is equal to 

(m \(-!) r 

R^z) = (Jldet(l - A l F.a.z)^ 1+L j (22) 

where a = (—l) p ,p , r, m and F is defined in section 1.5. 

Proof If we repeat the proof of the theorem 6 for <fi n instead we receive that 
R(4> n ) = (— l) r+pn det(l — F)( we suppose that Reidemeister numbers R(4> n ) are finite 
for all n).Last formula implies the trace formula for R{<p n ) : 

m 

R(<f) n ) = (-l) r+ P n ^(-l) i Tr (A l F) n 

i=0 

From this we have formula 22 immedeately by direct calculation as in theorem 14. 
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Corollary 4 Let the assumptions of theorem 17 hold. Then the poles and zeros of the 
Reidemeister zeta function are complex numbers which are reciprocal of an eigenvalue 
of one of the matrices 

A*(F) : A*{G) — ► A'(G) < i < rank G 



5.4.1 Functional equation 

Theorem 18 Let <p : G — > G be an endomorphism of a finitely generated torsion 
free nilpotent group G.Then the Reidemeister zeta function R$(z) has the following 
functional equation: 

^(^)=e 2 .^(,)(" 1 ) RankG . (23) 

where d = det F and e± are a constants in (U* . 

Proof Via the natural nonsingular pairing (A l F) <g> (A m ~ l F) — > (D the operators 
A m ~ l F and <i.(A l F) _1 are adjoint to each other. 

We consider an eigenvalue A of K l F. By theorem 17, This contributes a term 

(a -£)<-*")'-"' 



toi^(i). 

We rewrite this term as 



and note that | is an eigenvalue of A m l F. Multiplying these terms together we 
obtain, 

«*Q-(n n (^) M ') <_1) x^ w <-t. 

\»— 1 A«eSpec A'F / 

The variable z has disappeared because 

m m 

^(-l) 4 dimA l G = £(-1)*^ = 0. 

i=0 i=0 



5.5 Some conjectures for wider classes of groups 

For the case of almost nilpotent groups (ie. groups with polynomial growth, in view 
of Gromov's theorem [fffil ) we believe that some power of the Reidemeister zeta func- 
tion is a rational function. We intend to prove this conjecture by identifying the 
Reidemeister number on the nilpotent part of the group with the number of fixed 
points in the direct sum of the duals of the quotients of successive terms in the central 
series. We then hope to show that the Reidemeister number of the whole endomor- 
phism is a sum of numbers of orbits of such fixed points under the action of the finite 
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quotient group (ie the quotient of the whole group by the nilpotent part). The situ- 
ation for groups with exponential growth is very different. There one can expect the 
Reidemeister number to be infinite as long as the endomorphism is injective. This 
can be proved in the case of surface groups due to a theorem of C.Epstein (see J7J). 
He proves an estimate on numbers of geodesies, which when applied to the mapping 
torus of pseudo-Anosov map guarantees infinitely many loops which wrap around the 
mapping torus exactly once. This is equivalent (see section 4.1) to saying that the 
Reidemeister number is infinite. A rigid hyperbolic group has a finite outer auto- 
morphism group P3|. This implies that the Reidemeister number of some iteration of 
endomorphism equals to the number of usual conjugacy classes which is in this case 
infinite. We believe this conjecture for the following reason. Let G be a group with 
exponential growth and let 1(g) be the length of an element of G. Then one might 
expect that most of the time 

Kgx^g)- 1 ) > (1 + e)l(g) 
for g,x G G. This would imply that for fixed x, 

^{gx^gy 1 ■ l{gx<P(gY l ) <N}<#{geG: l(gx<t>( g y l ) < N/(l + e)} 
However since the group has exponential growth one can show : that this would imply 

^{gxm- 1 ■ KgxHg)- 1 ) < n} 



#{geG: l(gx<P(g)-') < N} 



as N — ► oo. 



If there were only finitely many twisted conjugacy classes then we could derive a 
contradiction by summing the left hand side of the above formula over a set of repre- 
sentatives x for the classes, and observing that this sum is always equal to 1. 

6 The Reidemeister and Nielsen zeta functions of 
a continuous map. 

Remark 2 Using lemma 6 we may apply all theorems of section 5 to the Reidemeister 
zeta functions of continuos maps. 



6.1 The Jiang subgroup and the Nielsen zeta function 

Theorem 19 Suppose that there is an integer m such that f™(ir) C I(f m ) .If L(f n ) ^ 
for every n > o ,then 

(— i) r 

N f (z) = R f (z) = (n.det(l - AY£ ® A.a.z)^ i+1 ) (24) 
If L(f n ) = only for finite number of n, then 

(k 
f[det(l-A7£® A.a.z)^ 1 
i=0 

(25) 

Where P(z) is a polynomial ,A, k,p, and r are as in theorem 14- 
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Proof 

If L(f n ) ^ for every n > o ,then formula (19) follows from theorems 9 and 14. If 
L(f n ) = 0, then N(f n ) = .If L(f n ) ^ 0, then N(f n ) = R(f n )(see proof of theorem 9 
).So the fraction Nf(z)/Rf(z) = exp(P(z)), where P(z) is a polynomial whose degree 
equal to maximal n, such that L(f n ) ^ 0. 

Corollary 5 Let the assumptions of theorem 19 hold. Then the poles and zeros of the 
Nielsen zeta function are complex numbers which are the reciprocal of an eigenvalue 
of one of the matrices 

\\f™®A.a) 



Corollary 6 Let I(id x ) = n -If the assumptions of theorem 19 about Lefschetz num- 
bers hold, then formulas (24) and (25) are valid. 



Corollary 7 Suppose that X is asphericaland f is eventually commutative. If the as- 
sumptions of theorem 19 about Lefschetz numbers hold, then formulas (24) and (25) 
are valid. 



6.2 Polyhedra with finite fundamental group and Nielsen 
zeta function 

Theorem 20 Let X be a connected, compact polyhedron with finite fundamental 
group it. Suppose that the action of ir on the rational homology of the universal cover 
X is trival, i.e. for every covering translation a G n , a* = id : H*(X, Q) — > H*(X, Q). 
If L(f n ) 7^ for every n > o ,then 

w = R ' iz) = m^Bry (26) 

If L(f n ) = only for finite number of n, then 

N,(z) = exp (P(z)) .R f (z) = exp (P(z)) (27) 

Where P(z) is a polynomial, B is defined in theorem 4 
Proof 

If L(f n ) ^ for every n > o ,then formula (26) follows from theorems 10 and 
15. If L(f n ) = 0, then iV(/ n ) = .If L(f n ) ^ 0, then N(f n ) = R(f n )(see proof of 
theorem 10). So the fraction Nf(z)/Rf(z) = exp(P(z)), where P(z) is a polynomial 
whose degree equal to maximal n, such that L(f n ) ^ 0. 

Corollary 8 Let X be a compact 1-connected polyhedron which is a rational homology 
n-sphere, where n is odd. Let n be a finite group acting freely on X and let X = X/tt. 
Then theorem 20 applies. 



Corollary 9 If X is a closed 3-manifold with finite ir, then theorem 20 applies. 
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Example 2(0) Let f : S 2 V S 4 — > S* 2 V S 4 to be a continuous map of the bouquet 
of spheres such that the restriction f / 54 = id$4. and the degree of the restriction 
f j 52 : S 2 — > S 2 equal to —2. Then L(f) = 0, hence N(f) = since S 2 V S 4 is simply 
connected. For k > 1 we have L(f k ) = 2 + (— 2) k 7^ , therefore N(f ) = l.From this 
we have by direct calculation that 

N f {z)=e X p{-z).j^. (28) 



Remark 3 We would like to mention that in all known cases the Nielsen zeta func- 
tion is a nice function. By this we mean that it is a product of an exponential of a 
polynomial with a function some power of which is rational. May be this is a general 
pattern; it could however be argued that this just reflects our inability to calculate the 
Nielsen numbers in general case. 

6.3 Nielsen zeta function in other special cases 

Theorem 17 implies 

Theorem 21 Let f be any continious map of a nilmanifold M to itself. If R(f n ) is 
finite for all n then 

/ m \ (-l) r 

N f {z) = R f {z) = \J[ det(l - k'F.a.zf-^j (29) 

where a = (—l) p ,p , r, m and F is defined in section 5.4- 
Theorem 12 implies 

Theorem 22 Let X be a compact surface of negative euler characteristic and f : 
X —>■ X is a pseudo-Anosov homeomorphism. Then 

in 

N f (z) = J] det(l - A iZ )-^ (30) 

where Ai and e, the same as in theorem 12. 
Theorem 13 implies 

Theorem 23 Suppose M is a orientable compact connected 3-manifold such that 
intM admits a complete hyperbolic structure with finite volume and f : M — >• M 
is orientation preserving homeomorphism. Then Nielsen zeta function is rational and 

N f (z) = L f {z) 
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6.4 Radius of convergence of the Nielsen zeta function 

We denote by R the radius of convergence of the Nielsen zeta function Nf(z), and by 
h(f) the topological entropy of continuous map /. Let h = inf{h(g) over all maps g 
of the same homotopy type as /}. 



Theorem 24 ( [|T8| ) For any continuous map f of any compact polyhedron X into 
itself the Nielsen zeta function has positive radius of convergence R and 

R > exp(-Ji) > 0. (31) 

In this section we propose another prove of positivity of R and give an exact 
algebraic lower estimation for the radius R using trace formulas (16) and (18) for 
generalised Lefschetz numbers. 

For any set S let ZjS denote the free abelian group with the specified basis S. The 
norm in /ZS is defined by 

|| hsi\\ :=Y^\ki\ e ^i (32) 

i i 

when the Sj in S are all different. 

For a ZZH-matrix A = (a y -), define its norm by \\A\\ := Eij ||ay||.Then we have 
inequalities ||AB|| < \\A |.||-B|| when A,B can be multiplied, and ||Tr A\\ < \\A\\ 
when A is a square matrix. For a matrix A = (a ij )iV(/ n ) in ZZS, its matrix of norms is 
defined to be the matrix A norm := (||a y -||) which is a matrix of non-negative integers. In 
what follows, the set S wiill be ir, H or H c .We denote by s(A) the spectral radius of 
A, s(A) = lim n (||74 n || |)n , which coincide with the largest modul of an eigenvalue of 
A. 

Theorem 25 For any continious map f of any compact polyhedron X into itself 
the Nielsen zeta function has positive radius of convergence R, which admits following 
estimations 

R > > (33) 

max,! \\zF d \\ 

and 

R > = > 0, (34) 

m&x d s(F2 orm ) 

where F d is the same as in section 4- 

Proof By the homotopy type invariance of the invariants we can suppose that / 
is a cell map of a finite cell complex.By the definition the Nielsen number N(f n ) is the 
number of non-zero terms in L n (f n )(see section 4). The norm HL^/" - )!! is the sum of 
absolute values of the indices of all the rz-orbits classes O n .It equals \\L n (f n ) \\ , the sum 
of absolute values of the indices of all the fixed point classes of f n , because any two 
fixed point classes of f n contained in the same n-orbit class O n must have the same 
index.From this we have N(f n ) < ||L w (/ n )|| = HMD II = II Ed(-l) d [Tr (^) n ]|| < 
£ d ||[Tr (zF d r\\\ < EjTr {zF d T\\ < E d ||(^) n || < E d II (zF d ) f(see P)The ra- 
dius of convergence R is given by Caushy-Adamar formula: 

— = limsup( — — — )» = limsup(A^(/ n ))™. 

R n n n 
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Therefore we have: 

1 1 

R = r > ^- > 0. 

Kmsup n (iV(/ n ))» max d \\zF d \\ 

Inequalities: 

mn < IMr) IHIM/") 11 = 11 EC-i)*^ W1II< Ell( Tr Wlll< 

d d 
^||Tr < ]TTr ((^) n )™ < ]T Tr ((^) norm ) ri < ]TTr ^F d ) norm ) n 

d d 

and the definition of spectral radius give estimation: 

R = r > L > 0. 

limsup n (iV(/ n ))» max d s(F d ™) 



Example 3 Let X be surface with boundary, and f : X — > X be a map.Fadell and 
Husseini j^J devised a method of computing the matrices of the lifted chain map for 
surface maps. Suppose {a\, a r } is a free basis forni(X). Then X has the homotopy 
type of a bouquet B of r circles which can be decomposed into one 0-cell and r 1-cells 
corresponding to the ai,and f has the homotopy type of a cellular map g : B —> B. 
By the homotopy type invariance of the invariants ,we can replace f with g in compu- 
tations. The homomorphism /* : i\\ [X) — > tti(X) induced by f and g is determined by 
the images bi = /^(aj),z = 1, ..,r. The fundamental group 7i~i(Ty) has a presentation 
TTi(Tf) =< ai, a r , z\aiZ = zb i: % — 1, .., r > .Let 

be the Jacobian in Fox calculus(see j^J). Then, as pointed out in the matrices of the 
lifted chain map g are 

F = (1),F 1 = D = (^). 
Now, we can find estimations for the radius R from (33) and (34). 



Remark 4 Let X be a compact connected surface and f : X —>■ X be a homeo- 
morphism.Let x{X) < O.By Thurston's classification theorem W\l f is isotopic to a 
homeomorphism <j) such that either (1) <fi is a periodic map; or (2) 4> is a pseudo-Anosov 
map with stretching factor A > l(see section 3.3.1);or (3) <fi is a reducible map, i.e. 
there is a system of disjoint simple closed curves in intX which is invariant by <fi and 
which has a4> -invariant tubular neighborhood U such that each component of X — U 
has negative euler characteristic and on each (^-component of X — U , cf) satisfies 1 
or 2. Then, as follows from \TBJ, radius R = \, where A > 1 is the largest stretching 



factor of the pseudo-Anosov piesesfif there is no pseudo-Anosov piece) A = 1. 
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7 Connection with Reidemeister Torsion 



7.1 Preliminaries 

7.1.1 Reidemeister torsion 

Like the Euler characteristic, the Reidemeister torsion is algebraically defined. Roughly 
speaking, the Euler characteristic is a graded version of the dimension, extending the 
dimension from a single vector space to a complex of vector spaces. In a similar way, 
the Reidemeister torsion is a graded version of the absolute value of the determinant 
of an isomorphism of vector spaces. Let d % : C l —>■ C t+1 be a cochain complex C* 
of finite dimensional vector spaces over W with C l = for % < and large i. If the 
cohomology H l = for all % we say that C* is acyclic. If one is given positive densities 
Aj on C % then the Reidemeister torsion r(C7*,Aj) G (0, oo) for acyclic C* is defined 
as follows: 

Definition 6 Consider a chain contraction 6 l : C l — > C* _1 ; ie. a linear map such 
that do<5 + 5od = id. Then d + 5 determies a map 

(d + 5) + : C + := ©C 2i -> C- := ©C 2i+1 and a map (d + 5)_ : C~ -> C+ . Since 
i/ie map (d + <5) 2 = zd + 5 2 zs unipotent, (d + 5) + must be an isomorphism. One defines 
r(C*,Ai) : = | det(d + 5) + | fsee J2]/j. 

Reidemeister torsion is defined in the following geometric setting. Suppose K is 
a finite complex and E is a flat, finite dimensional, complex vector bundle with base 
K. We recall that a flat vector bundle over K is essentially the same thing as a 
representation of 7Ti (fC) when X is connected. If p G K is a basepoint then one may 
move the fibre at p in a locally constant way around a loop in K. This 

defines an action of 7Tx(K) on the fibre E p of above p. We call this action 
the holonomy representation p : n — > GL(E p ). Conversely, given a representation 
p : 7i — > GL(V) of 7i on a finite dimensional complex vector space V, one may define 
a bundle E = E p = (K x V)/tt. Here is the universal cover of K, and 7r acts on 
K by covering tranformations and on V by p. The holonomy of E p is p, so the two 
constructions give an equivalence of flat bundles and representations of ix. 

If K is not connected then it is simpler to work with flat bundles. One then defines 
the holonomy as a representation of the direct sum of ix\ of the components of K. In 
this way, the equivalence of flat bundles and representations is recovered. 

Suppose now that one has on each fibre of E a positive density which is locally 
constant on K. In terms of pg this assumption just means | det pe \= 1- Let V denote 
the fibre of E. 

Then the cochain complex C l (K; E) with coefficients in E can be identified with 
the direct sum of copies of V associated to each i-cell o of K. The identification is 
achieved by choosing a basepoint in each component of K and a basepoint from each 
i-cell. By choosing a flat density on E we obtain a preferred density A* on C l (K, E). 
One defines the R-torsion of (K, E) to be t(K; E) = r(C*(K; E), A,) G (0, oo). 

7.1.2 Unitary dual of the direct sum of a free abelian and a finite group 

In this section let again the group G will be G = ZZ" 1 © F ,where F is a finite group 
and m is a natural number. Let : G — ► G be an endomorphism.One defines the 
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unitary dual G of G to be the set of all equivalence classes of irreducible, unitary 
representations of G. If p : G — > U(V) is a unitary representation of G on V then 
po(fi : G — > U (V) is also a representation, which we denote <p(p). If the representations 
Pi and p 2 are equivalent then representations (f>(pi) and 4>{p2) are also equivalent. 
Therefore the endomorphism <fi induces a dual map (j) : G — > G from the unitary dual 
to itself. Now consider the dual G for G = /Z r © F. This is the cartesian product of 
the duals of Z m and F: G = /Z m ® F. From this it follows that topologically the 
dual G is a uni! on of finitely many disjoint tori 



Lemma 10 jlq/ 7/0 any endomorphism of G where G is the direct sum of a finite 
group with a free abelian group, then 

R(<f>) = #Fix[$) 

7.2 The Reidemeister zeta Function and the Reidemeister 
Torsion of the Mapping Tori of the dual map. 

Let / : X — > X be a homeomorphism of a compact polyhedron X. 

Let Tf := (X x I)/(x, 0) ~ (f(x), 1) be the mapping tori of /. We shall consider 
the bundle p : Tf — » S 1 over the circle S* 1 . We assume here that is a flat, complex 
vector bundle with finite dimensional fibre and base S l . We form its pullback p*E over 
Tf. Note that the vector spaces H l {p^ 1 {b) , c) with b G S 1 form a flat vector bundle 
over S l , which we denote H l F. The integral lattice in H l {p~ 1 {b), H) determines a flat 
density by the condition that the covolume of the lattice is 1. We suppose that the 



bundle E <g> H l F is acyclic for all i. Under these conditions D. Fried plj has shown 
that the bundle p*E is acyclic, and we have 

r{T f ; P *E) = Y[r{S 1 ;E^H i F)^ i . (35) 

i 

Let g be the prefered generator of the group 7ri(S' 1 ) and let A = p(g) where p : 
n 1 (S 1 ) — > GL(V). Then the holonomy around g of the bundle E ® H l F is A <g> /*. 
Since r(S) =| det(J - A) \ it follows from (20) that 

r(Tf,p*E) = n I det(J — A® f*) \^ . (36) 

i 

We now consider the special case in which E is one-dimensional, so A is just a complex 
scalar A of modulus one. Then in terms of the rational function Lf(z) we have |2l| : 



T(Tf, P *E) = I] I det(J - A./*) I^" 1 ) =| L 7 (A) I" 1 (37) 



Theorem 26 Lei : G — > G be an automorphism of G , where G is the direct sum of 
a finite group with a finitely generated free abelian group, then 

r(T fP *E)=\L^X)r=\R,(a\)\^ r+ \ 

where A is the holonomy of the one-dimensional flat complex bundle E over S 1 , r and 
a are the constants described in theorem 5 . 
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Proof We know from lemma 10 that R(<f) n ) is the number of fixed points of the 
map <f) n . In general it is only necessary to check that the number of fixed points of 
n is equal to the absolute value of its Lefschetz number. We assume without loss of 
generality that n — 1. We are assuming that R(4>) is finite, so the fixed points of 
form a discrete set. We therefore have 

L(4>) — X! Index (0, x). 

Since is a group endomorphism, the trivial representation xq € G is always fixed. 
Let x be any fixed point of 0. Since G is union of tori Go, G t and is a linear map, 
we can shift any two fixed points onto one another without altering the map 0. This 
gives us for any fixed point x the equality 

Index (0, x) = Index (0, Xo) 

and so all fixed points have the same index. It is now sufficient to show that 
Index (0, xo) = ±1. This follows because the map on the torus 

: G — > G 

lifts to a linear map of the universal cover, which is an euclidean space. The index 
is then the sign of the determinant of the identity map minus this lifted map. This 
determinant cannot be zero, because 1 — must have finite kernel by our assumption 
that the Reidemeister number of is finite (if det(l — 0) = then the kernel of 1 — 
is a positive dimensional subspace of G, and therefore infinite). 

Corollary 10 Let f : X — > X be a homeomorphism of a compact polyhedron X . If 
tti(X) is the direct sum of a finite group with a free abelian group, then then 



r ( T ^) ; ^) = i L (^) (A) r=i^ A ) 



r + l 



where r and a are the constants described in theorem 9. 



8 Concluding remarks 

8.1 Reidemeister and Nielsen numbers and zeta functions 
modulo a normal subgroup 

In the theory of (ordinary) fixed point classes, we work on the universal covering 
space. The group of covering transformations plays a key role. It is not surprising 
that this theory can be generalized to work on all regular covering spaces. Let K be 
a normal subgroup of the fundamental group 7Ti(X). Consider the regular covering 
Pk '■ X/K — > X corresponding to K. A map fx '■ X/K — > X/K is called a lifting 
of / : X — > X if pk ° fx — f ° Pk- We know from the theory of covering spaces 
that such liftings exist if and only if f*(K) C K. If K is a fully invariant subgroup of 
TTi(X) ( in the sense that every endomorphism sends K into K) such as, for example 
the commutator subgroup of ni(X), then there is a lifting of any continuous map. 
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We may define the mod i^-Reidemeister and Nielsen numbers (see |[26|| ) and zeta 



functions (see |T2|]) and develop a similar theory for them by simply replacing X and 
7Ti(X) by X/K and tx\{X)/K in every definition, every theorem and every proof, 
since everything was done in terms 
of liftings and covering translations. 



8.2 Minimal dynamical zeta function 

8.2.1 Radius of convergence of the minimal zeta function 

In the Nielsen theory for periodic points, it is well known that N(f n ) is often poor as 
a lower bound for the number of fixed points of f n .A good homotopy invariant lower 
bound NF n (f), called the Nielsen type number for / n ,is defined in |26|]. Consider any 



finite set of periodic orbit classes {O kj } of varied period kj such that every essential 
periodic m-orbit class, m\n, contains at least one class in the set. Then NF n (f) is the 
minimal sum J2j kj for all such finite sets. Halpern(see |26|]) has proved that for all 



n NF n (f) = min{#Fix(g)\g has the same homotope type as / }. Recently, Jiang pT 
found that as far as asymptotic growth rate is concerned,these Nielsen type numbers 
are no better than the Nielsen numbers. 

Lemma 11 (Q27|]) 

limsup(iV(/ n ))» = limsup(iVF n (/))n (38) 

n n 

Let us consider minimal dynamical zeta function 



In paper it was proved that Mf(z) has positive radius of convergence R. Below 
we propose another prove of this fact and give an exast algebraic lower estimation for 
the radius R. 

Theorem 27 For any continious map f of any compact polyhedron X into itself the 
minimal zeta function has positive radius of convergence R,which admits following 
estimations 

R > exp(-/i) > 0, (39) 

R > > 0, (40) 

max d ||^F d || 

and 

R > L > 0, (41) 

max d s(F2 orm ) 

where Fd and h is the same as in section 6.1. 

Proof The theorem follows from Caushy-Adamar formula, lemma 11 and theo- 
rems 24 and 25. 
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8.3 Dold congruenses 

Definition 7 We shall say that a sequence of integers a(n) satisfies the Dold congru- 
ences iff for all n G N one has 

^2 fi(d)a(n/d) = mod n. 

d\n 

Dold || proved that the sequence L(f n ) of Lefschetz numbers has this property. 
The following easy lemma can be proved purely combinatorially: 



Lemma 12 Let X is any set and f : X — ► X any function. If for any n, f n has only 
finitely many fixed points in X then the sequence of numbers #Fix(f n ) satisfies the 
Dold congruences. Any sequence of natural numbers satisfying the Dold congruences 
arises in this way. 



In we have proved that if / : X — > X is eventually commutative map of com- 
pact polyhedron,or any self-map if tti(X) is finite, then the numbers R(f n ) satisfy the 
Dold congruences. 

Let g : M — > M be an expanding map [3(| of the orientable smooth compact 
manifold M.Then M is aspherical and is a K(jrx(M), 1), and 7Ti(M) is torsion free 
[Mj. According to Shub [|36| any lifting g of g has exactly one fixed point. From this 
and the covering homotopy theorem it follows that the fixed point of g are pairwise in- 
equivalent. The same is true for all iterates g n . Therefore N(g n ) = NF n (g) = #Fix(g n ) 
for all n.So the sequence of the Nielsen and Nielsen type numbers N(g n ) = NF n (g) 
of an expanding map satisfies the Dold congruences. 

For a hyperbolic endomorphism of torus T n and for a pseudo-Anosov homeomorphism 
of a compact surface we also have equalities N(g n ) = NF n (g) = #Fix(g n ) for all n 
(see [ 13 1 and p7|), and , therefore, the Dold congruences for the sequence of the Nielsen 



and Nielsen type numbers 



8.4 Open questions 

Question 1 For which spases and maps in the inequalities (33) - (34) does the equal- 
ity hold ? 

Sometime, the minimal zeta function coincide with the Nielsen zeta function,for 
example, for hyperbolic endomorphism of torus, for an expanding map of the ori- 
entable smoth manifold (see |H|).This motivate the following 



Question 2 For which spaces and maps minimal zeta function is a rational func- 
tion, when it is a meromorphic function? When does it have a functional equation? Which 
zeros and poles it has? 



Question 3 For which spases and maps we have trace formula for Nielsen type num- 
bers? When we have the Dold congruences for this numbers? When in the inequalities 
(37)- (38) does equalities hold? 
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The trace formulae which we recieve in this article appear to be very similar 
to formulae arising in thermodynamical formalism. The relation of the radius of 
convergence with the entropy (Theorem 24), Markov partition in the pseudo-Anosov 
case(Theorem 22) and the relation of the Reidemeister zeta function to Artin-Mazur 
zeta function on the unitary dual space(lemma 10) also indicate a connection with 
this theory. It is another open question to understand this connection. 
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